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ABSTRACT 

Alternate assessments based on alternate 
achievement standards (AA-AAS) are designed 
to measure the academic achievement of 
students with the most significant cognitive 
disabilities (SWSCDs). AA-AAS present unique 
measurement challenges because of the inherent 
need for individualization in item presentation 
and response, combined with expectations for 
rigorous levels of standardization. Additional 
measurement challenges are presented as states 
move toward implementation of growth models for 
accountability. This brief addresses some decisions 
that states face as growth models with AA-AAS are 
implemented. 


Background 

AA-AAS are designed to measure the academic 
achievement of students with the "most significant 
cognitive disabilities" exclusively as part of the U.S. 
Department of Education's effort to ensure that 
"schools are held accountable for the educational 
progress of students with the most significant 
cognitive disabilities" 1 . Students with intellectual 
disabilities, autism, or multiple disabilities make up 
the majority of the SWSCD population. 2 SWSCDs 
can be difficult to assess in a standardized manner. 3 
Measurement challenges for growth models include 
trend analysis discrepancies, distribution assumptions, 
compounded standard errors, and multiple scales. 4 
Additionally, significant data-related concerns for 
growth models include data system integrity, missing 
data, student mobility, student attrition, and scaling 
difficulties. 5 These challenges are confounded with 
even more factors for AA-AAS, including: (a) eligibility 
concerns, (b) lack of a comparison group for certain 
grade levels, (c) retention (and/or lack of grade 
level consistency), (d) within-group variability, and 
(e) reporting levels. The measurement difficulties 


inherent within AA-AAS may also lead to different 
decisions made across tests when AA-AAS results are 
used in statewide accountability growth models and 
adequate yearly progress (AYP) determinations. 

Transition Matrix Approach 

In the Transition Matrix (TM) approach, 
student growth can be depicted as changes in 
percentages and/or frequencies of students at various 
performance standard levels (e.g.. Does Not Yet Meet, 
Nearly Meets, Meets, Exceeds in Oregon). TMs also 
can award points to students in multiple ways. 

In order to successfully represent growth of SWSCDs 
within existing accountability systems, we present a 
model with points given for students who perform 
at a higher level from one year to the next, and 
points subtracted for students who performed at a 
lower level from one year to the next. No points are 
awarded for students who maintain performance 
levels from one year to the next (with the exception of 
the Exceeds level). 

AA-AAS is typically developed on scales 
that preclude comparisons across different state 
assessments. The TM approach is ideal for AA-AAS 
applications, as it allows for scores from tests on 
different scales to be aggregated on a common scale 
of change. An important assumption in documenting 
this type of change, however, is that appropriate 
and articulated procedures have been adopted and 
followed in setting standards and proficiency cut 
scores. 

Sample and Measure 

This longitudinal, grade-level comparison study 
used data from the Oregon Extended Assessment 
(ORExt) in reading for SWSCDs in Grades 3-8 from 
2008-2009. Of the 6,722 students in the 2008/09 
dataset, 2,933 students with test data for both years 
were used in the TM analyses. 
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Change in Performance Categories: Four-Category 


m 1200 

b 


S wo 



■ 7 


Figure 1. Clustering of student growth near 'O', demonstrating no growth from one year to the next in 
reading for most students. 


Results 

We constructed 
cross tabulation 
tables comparing the 
frequencies of each 
of four performance 
categories in order to 
generate transition 
matrices for the 
2008/09 transition 
years. These are 
longitudinal analyses, 
comparing the same 
students from one 
year's summative 
assessment to the 
next year's summative 
assessment. The 
four performance 
categories were: 

Does Not Yet Meet 
(DNYM), Nearly Meets (NM), Meets (M), and Exceeds 
(E). Students earned a point for moving up one 
performance level and lost a point for moving down 
one performance level. For example, a student who 
moved up one performance level from DNYM to NM 
generated a +1. A student who fell a performance level 
from E to M generated a -1. A student earned zero 
points for remaining at the same performance level, 
except those who remained at E, who earned one 
point. Results across Grades 3-8 for all students in the 
2008/09 transition years are presented in Figure 1. It 
appears that most students are maintaining current 
levels of performance from one year to the next with 
the four-category analyses. 

The majority of students maintained their 
performance level (1756/2874 = 61%). A total of 475 
students dropped one or more performance levels 
(475/2874 = 16.5%), while 643 students advanced 
one or more performance levels (643/2874 = 22%). 

The overall trend is upward, meaning that more 
students gained a performance level (643) than lost a 
performance level (475). 

A more discrete analysis was also conducted 
using RIT-scaled scores for this data set. We were 
interested in looking at how shifting the number of 
performance categories might affect outcomes. An 
arbitrary RIT-score range of 30 points for each category 
was selected, resulting in seven categories. Results of 


this analysis are presented below. 

The seven-category longitudinal model 
shows that the majority of students are indeed 
growing and not merely maintaining previous levels 
of performance, as found with the four-category 
longitudinal model. With the four-category model, 
we would be led to believe that 61% of our students 
are not growing. With the seven-category model, we 
see that approximately 55.5% of our students grew in 
a discrete manner, but likely not enough to put them 
over the bar to earn the next performance level. 
Overall, the trend also remains positive, with more 
students increasing (1,588) compared to decreasing 
(1,146). 

Grade level differences are also apparent 
between the two models. In fact, 6th grade went 
from the highest performing grade level in the four- 
category approach (10.9% of students performing 
one or more performance levels lower from 2008 
to 2009, while 33% of students advanced one or 
more performance levels) to the lowest performing 
grade level in the seven-category approach (63.2% of 
students performed one or more performance levels 
lower from 2008 to 2009, while 30.6% of students 
advanced one ore more performance levels) simply 
due to the number of performance levels included 
in the calculations and the shapes of the respective 
distributions. 
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Figure 2. Students who are growing slightly from one year to the next in reading. 


Future directions 

The TM approach is flexible and feasible to 
implement with existing status-based performance 
structures. While this model is efficient and appears to 
hold some promise, it is shared not as a standard for 
the AA-AAS field to adopt, but to illustrate some of the 
policy and measurement challenges the field faces in 
implementing growth models in a robust manner for 
SWSCDs. 

With AA-AAS, only a limited range of 
possibilities can be investigated. In the end, the field 
needs to define at the school, district, and state levels 
how much growth is enough. Ideally, future federal 


and state policies will address the needs surrounding 
growth models to support their implementation, as 
the move from status-based models toward growth 
models progresses. It appears unlikely that states 
will be in a position to implement valid growth 
models, even TMs, for SWSCDs until they have 
addressed at least eligibility concerns, participation, 
retention, within-group variability, and reporting 
level. Additional measurement concerns that are 
common across assessments, such as data system 
integrity, missing data, student mobility, student 
attrition, trend analysis discrepancies, distribution 
assumptions, compounded standard errors, and 
scaling difficulties will need to be addressed as well. 
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